Deep Cytometry: Deep learning with Real-time Inference in Cell Sorting and Flow Cytometry

Deep learning has achieved spectacular performance in image and speech recognition and synthesis. It outperforms other machine learning algorithms in problems where large amounts of data are available. In the area of measurement technology, instruments based on the photonic time stretch have established record real-time measurement throughput in spectroscopy, optical coherence tomography, and imaging flow cytometry. These extreme-throughput instruments generate approximately 1 Tbit/s of continuous measurement data and have led to the discovery of rare phenomena in nonlinear and complex systems as well as new types of biomedical instruments. Owing to the abundance of data they generate, time-stretch instruments are a natural fit to deep learning classification. Previously we had shown that high-throughput label-free cell classification with high accuracy can be achieved through a combination of time-stretch microscopy, image processing and feature extraction, followed by deep learning for finding cancer cells in the blood. Such a technology holds promise for early detection of primary cancer or metastasis. Here we describe a new deep learning pipeline, which entirely avoids the slow and computationally costly signal processing and feature extraction steps by a convolutional neural network that directly operates on the measured signals. The improvement in computational efficiency enables low-latency inference and makes this pipeline suitable for cell sorting via deep learning. Our neural network takes less than a few milliseconds to classify the cells, fast enough to provide a decision to a cell sorter for real-time separation of individual target cells. We demonstrate the applicability of our new method in the classification of OT-II white blood cells and SW-480 epithelial cancer cells with more than 95% accuracy in a label-free fashion

Effects of Anacetrapib in Patients with Atherosclerotic Vascular Disease

BACKGROUND: Patients with atherosclerotic vascular disease remain at high risk for cardiovascular events despite effective statin-based treatment of low-density lipoprotein (LDL) cholesterol levels. The inhibition of cholesteryl ester transfer protein (CETP) by anacetrapib reduces LDL cholesterol levels and increases high-density lipoprotein (HDL) cholesterol levels. However, trials of other CETP inhibitors have shown neutral or adverse effects on cardiovascular outcomes. METHODS: We conducted a randomized, double-blind, placebo-controlled trial involving 30,449 adults with atherosclerotic vascular disease who were receiving intensive atorvastatin therapy and who had a mean LDL cholesterol level of 61 mg per deciliter (1.58 mmol per liter), a mean non-HDL cholesterol level of 92 mg per deciliter (2.38 mmol per liter), and a mean HDL cholesterol level of 40 mg per deciliter (1.03 mmol per liter). The patients were assigned to receive either 100 mg of anacetrapib once daily (15,225 patients) or matching placebo (15,224 patients). The primary outcome was the first major coronary event, a composite of coronary death, myocardial infarction, or coronary revascularization. RESULTS: During the median follow-up period of 4.1 years, the primary outcome occurred in significantly fewer patients in the anacetrapib group than in the placebo group (1640 of 15,225 patients [10.8%] vs. 1803 of 15,224 patients [11.8%]; rate ratio, 0.91; 95% confidence interval, 0.85 to 0.97; P=0.004). The relative difference in risk was similar across multiple prespecified subgroups. At the trial midpoint, the mean level of HDL cholesterol was higher by 43 mg per deciliter (1.12 mmol per liter) in the anacetrapib group than in the placebo group (a relative difference of 104%), and the mean level of non-HDL cholesterol was lower by 17 mg per deciliter (0.44 mmol per liter), a relative difference of -18%. There were no significant between-group differences in the risk of death, cancer, or other serious adverse events. CONCLUSIONS: Among patients with atherosclerotic vascular disease who were receiving intensive statin therapy, the use of anacetrapib resulted in a lower incidence of major coronary events than the use of placebo. (Funded by Merck and others; Current Controlled Trials number, ISRCTN48678192 ; ClinicalTrials.gov number, NCT01252953 ; and EudraCT number, 2010-023467-18 .)

Machine Learning in Label-free Phenotypic Screening

High-throughput multivariate sensing is essential for high-content cellular phenotypic screening, where large volume of data for detection and accurate classification of rare events are often required. Here, we introduce time-stretch quantitative phase imaging (TS-QPI), a high-throughput label-free imaging flow cytometer developed for big data acquisition and analysis in phenotypic screening. TS-QPI is able to capture quantitative optical phase and intensity images simultaneously, enabling high-content cell analysis, cancer diagnostics, personalized genomics, and drug development.We further developed a complete machine learning pipeline that performs optical phase measurement, image processing, feature extraction, and classification. Multiple biophysical features such as morphological parameters, optical loss characteristics, and protein concentration are measured on individual cells. These biophysical measurements form a hyperdimensional feature space in which supervised learning is performed for cell classification. The technology is in clinical testing for blood screening and circulating tumor cell detection, as well as studying lipid accumulating algal strains for biofuel production. By integrating machine learning with high-throughput quantitative imaging, this system achieves record high accuracy in label-free cellular phenotypic screening and opens up a new path to data-driven diagnosis.Furthermore, we demonstrated, for the first time, real-time image compression performed in the optical domain to solve the big data challenge created by ultrafast measurement systems. Many ultrafast and high-throughput data acquisition equipment, including TS-QPI, produce a large volume of data in a short time, e.g. tens of Terabytes per hour. Such a data volume and velocity place a burden on data acquisition, storage, and processing and calls for technologies that compress images in optical domain and in real-time. As a solution, we have experimentally demonstrated warped time stretch, which offers variable spectral-domain sampling rate, as well as the ability to engineer the time-bandwidth product of the signal's envelope to match that of the data acquisition systems. We also show how to design the kernel of the transform and specifically, the nonlinear group delay profile governed by the signal sparsity. Such a kernel leads to smart detection with nonuniform spectral resolution, having direct utility in improvement of data acquisition rate, real-time data compression, and enhancement of ultrafast data capture accuracy

Data from: Optical data compression in time stretch imaging

Time stretch imaging offers real-time image acquisition at millions of frames per second and subnanosecond shutter speed, and has enabled detection of rare cancer cells in blood with record throughput and specificity. An unintended consequence of high throughput image acquisition is the massive amount of digital data generated by the instrument. Here we report the first experimental demonstration of real-time optical image compression applied to time stretch imaging. By exploiting the sparsity of the image, we reduce the number of samples and the amount of data generated by the time stretch camera in our proof-of-concept experiments by about three times. Optical data compression addresses the big data predicament in such systems

Artificial intelligence in label-free microscopy: biological cell classification by time stretch

This book introduces time-stretch quantitative phase imaging (TS-QPI), a high-throughput label-free imaging flow cytometer developed for big data acquisition and analysis in phenotypic screening. TS-QPI is able to capture quantitative optical phase and intensity images simultaneously, enabling high-content cell analysis, cancer diagnostics, personalized genomics, and drug development. The authors also demonstrate a complete machine learning pipeline that performs optical phase measurement, image processing, feature extraction, and classification, enabling high-throughput quantitative imaging that achieves record high accuracy in label -free cellular phenotypic screening and opens up a new path to data-driven diagnosis. • Demonstrates how machine learning is used in high-speed microscopy imaging to facilitate medical diagnosis; • Provides a systematic and comprehensive illustration of time stretch technology; • Enables multidisciplinary application, including industrial, biomedical, and artificial intelligence

Optical data compression in time stretch imaging.

Time stretch imaging offers real-time image acquisition at millions of frames per second and subnanosecond shutter speed, and has enabled detection of rare cancer cells in blood with record throughput and specificity. An unintended consequence of high throughput image acquisition is the massive amount of digital data generated by the instrument. Here we report the first experimental demonstration of real-time optical image compression applied to time stretch imaging. By exploiting the sparsity of the image, we reduce the number of samples and the amount of data generated by the time stretch camera in our proof-of-concept experiments by about three times. Optical data compression addresses the big data predicament in such systems

Illustration of warped stretch transform in imaging.

<p>(a) The field of view consists of a cell against the background such as a flow channel or a microscope slide. Illumination by an optical pulse that is diffracted into a one-dimensional rainbow maps one dimension of the space into the optical spectrum. The other dimension is scanned by the cell flow through the rainbow. In the conventional time stretch imaging (STEAM), the spectrum is linearly mapped into time using a dispersive optical fiber with a linear group delay. The temporal waveform is then sampled by a digitizer with fixed sampling rate resulting in uniform spatial sampling. But uniform spatial sampling generates superfluous data by oversampling the sparse peripheral sections of the field of view. (b) The human vision is a form of warped imaging system where high sampling resolution is needed in the central vision while coarse resolution can be tolerated in the peripheral vision. (c) Similar functionality can be achieved in STEAM by using a nonlinear group delay profile in the spectrum-to-time mapping process resulting in a nonuniform sampling of the line image, assigning more pixels to the information-rich central part of the field of view and less to the low-entropy peripherals. (d) The reconstruction is similar to anamorphic art where the drawn shape is a stretched and warped version of the true object yet the viewer sees the true object upon reflection from a curved mirror. In our system, this unwarping operation is a nonlinear mapping using the inverse space-to-frequency-to-time mapping transfer function.</p